Robustness in speech quality assessment and temporal training expiry in mobile crowdsourcing environments

Following up on prior work on assessment of quality of speech in laboratory environments, this paper introduces two recently released mobile crowdsourcing paradigms. In comparison to web-based crowdsourcing, mobile crowdsourcing is carried out on smartphones or tablets in the field. Firstly, because involved hardware such as headphones cannot be known in this paradigm, we focus on the effect of mobile crowdsourcing on the assessment of quality of speech using quality degradation types which are described for the model in ITU-T Rec. P.863. As a result, indicators for degradation types that can reliably be assessed in mobile crowdsourcing paradigms are presented for the first time. This reliability is interpreted as robustness towards crowdsourcing assessment environments. Secondly, because working times, pauses and work fragmentation cannot be controlled, we introduce and focus on the analysis of temporarily expiring training certificates as qualifications. Accordingly, we design our study to automatically issue re-training job instances by timeouts, aiming at re-conditioning distracted or oblivious crowd workers. Results indicate a clear improvement in terms of correlation to laboratory test results, when applying the proposed training expiry. Eventually, the indicators presented contribute to build up preliminary guidelines on practical execution of quality assessment using mobile crowdsourcing.